Technical Report Spatiotemporal Regularity Flow (SPREF): Its Estimation and Applications

Feature selection and extraction is a key operation in video analysis for achieving higher level of abstraction. In this report, we introduce a general framework to extract a new spatiotemporal feature that represents the directions in which a video is regular, i.e. the pixel appearances change the least. Explicit modeling of these directions is very useful and desired by many applications. We propose to model the directions of regular variations with a 3D vector field, which is referred to as SPatiotemporal REgularity Flow (SPREF). SPREF vectors are designed to have three cross-sectional parallel components Fx, Fy, and Ft for convenient use in different applications. They are estimated using all the frames simultaneously by minimizing an energy functional formulated according to its definition. In this report, we first introduce translational SPREF (T-SPREF) and then extend our framework to affine SPREF (A-SPREF). Then we demonstrate the use of SPREF in a few applications, including object removal, video inpainting, and video compression. The promising experimental results prove the usefulness and versatility of this new framework.